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Introduction 


•  Motivation 

•  The  Interface  Issue 


THI 


•  Problem  Statement 

—  "When  using  a  virtual  desktop  simulation  for 
training,  do  commercial  head  tracking  devices  and 
game  controllers  improve  training  effectiveness?" 
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Background 

•  Literature  Review 

—  Evolution  of  game  controllers 
-  Use  of  Game  controllers  outside  of  video  games 
—  Personnel  computers  verses  console  video  games 

•  Virtual  Battlespace  2  (VBS2™) 

•  Sony  PlayStation  3  game  controller 

•  Natural  Point  TrackIR  5 


INI 


4 


Methodology 


•  Phases 

1.  Define  the  scenario  and  determine  how  to 
measure  performance 

2.  Design  and  develop  the  scenario  in  VBS  2™ 

3.  Design  data  collection  system 

4.  Conduct  simulation  interface  experiment 

a.  Pilot  Study 

b.  Defense  Language  Institute  (DU)  Study 

5.  Evaluate  results  of  the  simulation  interface 
experiment 
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Methodology  ;o 

m 

•  Phase  1  -  Define  the  scenario  and  determine  :t/) 
how  to  measure  performance 

—  What  is  important? 

—  Move,  Shoot,  &  Communicate 
-  Mounted,  dismounted,  or  both? 

—  Metrics 

•  Time 

•  Accuracy 
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Methodology  ;o 
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•  Phase  2  -  Design  and  develop  the  scenario  in  :t/) 
VBS  2™ 
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1.  Start  point 

2.  Serpentine 

3.  Battle  Position  1 

4.  Urban  Driving 

5.  Battle  Position  2 

6.  Dismount  Point 

7.  Shoot  /No  Shoot  Lane 

8.  Ramps  and  Mazes 

9.  Small  Arms  (M4)  Range 


& 
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Evaluation 

Course 


Methodology 

Phase  3  -  Design  data  collection  system 

-  Automated  Scoring  System 

•  JAVA  Jar  application  written  to  capture  VBS  2™  DIS  data 
packets  over  network 

•  Time 

—  Crossing  trigger  lines  times 
—  Target  hit  times 

•  Accuracy 

—  Number  of  rounds  fired 

—  After  Action  Review  Scoring  System 

•  Accuracy 

-  Running  off  road,  crashing  vehicle,  falling  off  ramps 
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Methodology 

•  Phase  4a  -  Conduct  Pilot  Study 

—  16  MOVES  students  conducted  two  session 

•  First  session  with  keyboard  and  mouse 

•  Second  session  with  PS3  game  controller 

•  TrackIR  5  was  excluded 

—  Lessons  learned 

•  Fine  tuning  of  surveys,  briefs,  and  evaluation  courses 

•  Keyboard  and  game  controller  function  mapping 

•  AAR  recording  and  VBS2  Version  Control 

•  Automated  scoring  application 
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Methodology  ;o 
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•  Phase  4b  -  Defense  Language  Institute  Study  :(/) 

-  53  total  participants,  31  for  PS3  Study,  22  for  TrackIR 
Study 

—  All  participants  were  Army  Soldiers  (E-l  thru  E-6) 
enrolled  in  language  training  at  DU 

-  The  average  age  was  24.4 

—  6  of  the  53  were  female 

—  34  of  the  53  (64%)  identified  themselves  as 
"Gamers" 
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Results 


•  Phase  5  -  Evaluate  results  of  the  simulation 
interface  experiment 
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•  The  PS3  game  controller  outperformed  the  standard 
keyboard  and  mouse. 

•  The  PS3  improved  overall  performance,  mounted  shooting, 
dismounted  shooting,  and  shoot  house  event  scores. 


Normalized  performance  Scores 
(lower  is  better) 


Mounted  Mounted  Dismounted  Dismounted  Shoot 
Moving  Shooting  Moving  Shooting  House 
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t-Test:  Paired  two  sample  for  Means 

(Matched) 


Event 

PS3  Game 

Controller 

(Mean) 

PS3  Game 

Control 

(Variance) 

Keyboard  and 
Mouse 
(Mean) 

Keyboard  and 
Mouse 
(Variance) 

Two  tailed  P 
Value 

Total  Score 

16.2 

64.7 

21.1 

55.9 

0.0004 

Mounted 

Movement 

3.4 

4.1 

3.2 

4.2 

0.615 

Mounted 

Shooting 

2.4 

6.5 

3.9 

7.9 

0.036 

Dismounted 

Movement 

2.3 

3.8 

2.5 

3.0 

0.314 

Dismounted 

Shooting 

2.8 

3.2 

4.3 

3.6 

0.004 

Shoot  House 

2.15 

3.13 

3.5 

5.16 

0.0002 
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Results 


•  The  NaturalPoint  TrackIR  5  did  not  effect 
training  performance. 

•  Preferences: 

•  56%  preferred  using  the  PS3  game  controller 

•  54%  preferred  using  the  TrackIR  5  head  tracker 

•  76%  preferred  using  imagery  over  maps  for 
navigation. 
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Total  Score  for  Study  Total  Score  for  Study 


Methodology  -  Regression 


Age  vs  PS3  Score  Line  Fit  Plot 
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Age  vs  KB  Score  Line  Fit  Plot 
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♦  Total  Score  for  Study 
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Study 
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AN  OVA 


df 

SS 

MS 

F 

Significance 

F 

Regression 

l 

137.00 

137.00 

2.20 

0.15 

Residual 

29 

1804.59 

62.23 

Total 

30 

1941.59 

Standard 

Coefficients 

Error 

tStat 

P-value 

Lower  95% 

Intercept 

26.83 

7.24 

3.71 

0.00 

12.03 

Age 

-0.41 

0.28 

-1.48 

0.15 

-0.99 

ANOVA 

df 

55 

MS 

F 

Significance 

F 

Regression 

1 

0.98 

0.98 

0.02 

0.90 

Residual 

29 

1676.47 

57.81 

Total 

30 

1677.45 

Coefficients 

Standard 

Error 

tStat 

P-value 

Lower  95% 

Intercept 

22.01 

6.98 

3.15 

0.00 

7.74 

Age 

-0.04 

0.27 

-0.13 

0.90 

■^0.59 

Age 


Total  Score  for  Study  Total  Score  for  Study 
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Methodology  -  Regression 


Years  on  PC  Games  Line  Fit  Plot 


40 

35  ^ 
30 
25 
20 
15 
10 
5 
0 


ANOVA 


df 

55 

MS 

F 

Significance  F 

♦ 

Regression 

1 

566.20 

566.20 

14.78 

0.00 

♦ 

Residual 

29 

1111.25 

38.32 

■ 

♦ 

♦  Total  Score  for 

Total 

30 

1677.45 

$ 

▲ 

♦ 

Study 

♦♦ 

■  Predicted  Total 
Score  for  Study 

Coefficients 

Standard 

Error 

tStat 

P-value 

Lower  95% 

Intercept 

25.71 

1.63 

15.75 

0.00 

22.37 

10 

20 

30 

Years  on  PC 

Games 

-0.48 

0.13 

-3.84 

0.00 

-0.74 

Years  on  PC  Games 


Years  on  CS  Games  Line  Fit  Plot 


♦  Total  Score  for 
Study 

■  Predicted  Total 
Score  for  Study 


ANOVA 


df 

55 

MS 

F 

Significance  F 

Regression 

1 

232.78 

232.78 

3.95 

0.06 

Residual 

29 

1708.81 

58.92 

Total 

30 

1941.59 

Standard 

Coefficients 

Error 

tStat 

P-value 

Lower  95% 

Intercept 

19.73 

2.21 

8.93 

0.00 

15.21 

years  on  CS 
Games 

-0.31 

0.16 

-1.99 

0.06 

-0.64 

Years  on  CS  Games 
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Total  Score  Difference  (KB  -  PS3) 


Methodology  -  Regression 


Age  and  Performance  Line  Fit  Plot 


Age 


♦  Total  Score  Difference 
(KB  -  PS3) 

■  Predicted  Total  Score 
Difference  (KB  -  PS3) 


ANOVA 


df 

SS 

MS 

F 

Significance  F 

Regression 

1 

222.46 

222.46 

5.43 

0.027 

Residual 

29 

1187.89 

40.96 

Total 

30 

1410.35 

Standard 

Coefficients 

Error 

tStat 

P-value 

Lower  95% 

Intercept 

-9.27 

6.15 

-1.51 

0.14 

-21.86 

Age 

0.56 

0.24 

2.33 

0.027 

0.07 
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Recommendations 


•  Commander  and  BCTC  Directors  consider 
adding  PS3  game  controllers  as  an  additional 
user  input  device. 

•  Commander  and  BCTC  Directors  do  not 
consider  adding  TrackIR  5  as  an  additional 
user  input  device. 
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Questions? 


7/14/2010 
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Results 
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Gamer  stats 

—  34  of  the  53  (64%)  identified  themselves  as 
"Gamers" 

—  The  "non  gamers"  averaged  4.6  years  of  experience 
playing  video  games  at  2.1  hours  per  week 

—  The  "Gamers"  averaged  10.4  years  of  experience 
playing  PC  Games  at  8.4  hours  per  week 

—  The  "Gamers"  averaged  12.2  years  of  experience 
playing  console  video  games  at  9.7  hours  per  week 

—  Many  commented  they  would  play  more  if  not  for 
learning  a  second  language 
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